It depends on how the physics engine has been integrated into your game/engine. The book "Game Coding Complete" shows how to isolate a 3rd-party physics engine so you can switch implementations.
Of course, every physics engine behaves a little differently from the others, so, if you game is physics centered (pinball, racing), switching implementation might result in a slightly different game - but having the core of your game depends on the implementation details of some 3rd-party library might not be a good idea anyway.
In general, the more you sprinkle your code with dependencies to 3rd party libraries, the less control you have over the resulting product.
Your point still stands, though, for some very low-level "utility" libraries like boost or the STL, or any standard library replacement , and more generally, libraries holding "vocabulary" types.
Of course, every physics engine behaves a little differently from the others, so, if you game is physics centered (pinball, racing), switching implementation might result in a slightly different game - but having the core of your game depends on the implementation details of some 3rd-party library might not be a good idea anyway.
In general, the more you sprinkle your code with dependencies to 3rd party libraries, the less control you have over the resulting product.
Your point still stands, though, for some very low-level "utility" libraries like boost or the STL, or any standard library replacement , and more generally, libraries holding "vocabulary" types.