While sorting posts by time and votes makes a lot of sense, sorting comments in that same manner seems wrong, perhaps even counter-intuitive (e.g., a new, perhaps inconsenqeuncial, comment becomes prominent just by merrit of being new). Shouldn't comments be sorted simply by votes?
The two considerations seem to be: path dependency and natural conversation.
1-2-3-4 several points being made (natural)
184.108.40.206.1 several responses to the first point (never get to point 2)
4-2-1-3 Points no longer relate to each other in context (must be longer, self-contained)
Ultimately, you replace one type of "getting sidetracked" with another: "path dependency". Either way, the contextual information content is what you are trying to preserve. Without "exposure", for example, new comments in long conversations won't accrete upvotes to rise to the top, regardless of quality.