I am a mathematician, too and trying to learn physics in my spare time. I don't like the 1000 pages tomes targeted for undergraduates, the Halliday book is well-liked, but since you already have a degree in mathematics, you may get a bit bored.
I like the No-Nonsense Series by Jakob Schwichtenberg a lot. I worked through the Classical Mechanics book already and found that it teaches the conceptual framework very nicely. I plan to go through the Electrodynamics and Quantum Mechanics books in the series and then work through some other books with exercises (there are no exercises in the Schwichtenberg books, which usually is a no-go for me, but I am reading them to get the concepts in an uncluttered fashion.).
The best classical mechanics book IMO is the one by John R. Taylor, I have worked through 2/3 of it with nearly all exercises years ago, I can heartily recommend it.
For Electrodynamics I would recommend the book by Griffiths and for QM there is a very promising new book by Barton Zwiebach which has been created based on a course he is giving at MIT.
I'd like to understand General Relativity, and do not care about Quantum Physics. I have found this post to be helpful. Hopefully it can help you to some degree, too.
Is Fundamentals of Physics by Halliday et al a good starting point for me?