I don't think you are misinterpreting anything. I think those glances are part of survival instinct, assessing whether the moving object they registered out of the corner of their eyes (you), that is getting near them, is dangerous or if it can be safely ignored.
