The problem is that you an use words like "go" and "direction" in a situation where they make sense. In the incipient phase of the Universe the space-time did not exist as we can perceive it with our brains, themselves based on physical and chemical properties of the mature Universe - so would be impossible to "imagine" that there is no "north" because there is no direction available.
Kant deducted that our conscience and brains are born with the sense of space-time and we cannot override this. The only way in which we can access this ideas is only through highend mathematics and AIs that just need to do a job, but not to explain to our limited brains how.
Kant may have deducted that, but he's talking about a neurobiological phenomenon born from evolutionary pressures in a very limited environment.
Michelson–Morley mesured, and Einstein explained, how speed of light, not time, is constant through the Universe. To the point of questioning simultaneity.
If we've already got non-constant time, then the concept of a time origin prior to which there is no sensible physical description is at least plausible, if not something I can, spacial geometric puns pardoned, wrap my brain around in the same sense that there's a limit to "northness" on a spheroid.
Kant deducted that our conscience and brains are born with the sense of space-time and we cannot override this. The only way in which we can access this ideas is only through highend mathematics and AIs that just need to do a job, but not to explain to our limited brains how.