For me it wouldn't necessarily be a deal-breaker, but I would demand a significant premium on salary or cash signing bonus for the risk of not knowing the group and manager I'm going to be working with, or the work that I'm going to be doing. That seems like a pretty bad incentive for the hiring company too.
