Give it a shot; there's a fair number of non-mathy digressions that are worth reading, and you sound like you have the background to handle the mathier parts.
Of course different books help most for different people. This one's not the best if you want to pass a course and put it behind you. I think it's very much worth checking out if you want to be challenged and inspired to tackle problems for yourself and perhaps pick up a bit of how Feynman thought. There are very few other introductory texts by physicists of his stature: Matter and Motion by Maxwell, and, well, anything else? (Not counting popularizations; it's a fuzzy distinction, but these are securely on the textbook side.)