Either they were borrowed from the brokerage firms inventory or they were borrowed on margin from another clients account. If they're borrowed from the firms inventory, it's unlikely the short seller will be affected, because the firm will just have you borrow against different shares in their same pool of inventory. If they have no more left, they can borrow from another firm. In the very unlikely scenario that the firm decides to not hold ANY more shares of that company, then you may get called on the stock.
In the second scenario, if you're borrowing from another client, however, you may be at risk of getting a margin call -- in which case you would need to purchase the shares on the open market at their current price and return them to the lender.