Sure it's possible, and I don't think that's part of the security model. If you're letting app/library "A" access your location, you're trusting it. If you're worried that it might send that data to your worst enemy, you should disable access for library A.
