ODE seemed promising when I looked at it several years ago. Bullet is also quite popular. Searching for a comparison of the two led me to An Evaluation of Open Source Physics Engines for Use in Virtual Reality Assembly Simulations:
The authors, who were already using Bullet, concluded that "Newton and PhysX would be valuable candidates to compete with Bullet for integration in our current simulation environment."
Bullet seems to be the most mature of the game-oriented physics engines, but its story for those of us not using C++ is significantly less compelling. Bullet 2 doesn't have much of a C API, while Bullet 3 requires a high-end desktop card, considerably narrowing your customer base. Guess I'll have to re-evaluate the alternatives or resign myself to a truckload of shim C++.
C++'s generally poor interoperability has been annoying me since the 90s. :-(
http://elib.dlr.de/79462/1/74320346.pdf
The authors, who were already using Bullet, concluded that "Newton and PhysX would be valuable candidates to compete with Bullet for integration in our current simulation environment."