Well, yes, you need to know the semantics of the relationship, but not for parsing, because he content-type plus, where needed, internal data within the resource representation more specifically identifying the subtype should tell you what you need to know about handling the representation.
If you’re interacting with the api then it’s extremely likely you’re not going to write code that will automatically follow reference links and attempt to parse them dynamically.
You will need to know what fields are in the end point how they represent data (iso 8601 for dates? Unix timestamps?).
You’re substituting documentation with asking the api consumer to explore and discover and assume.
But you've documented how your API works anyway, right?