The gravity well difference is a thing but isn't straightforward. You need twice as much delta-v to take off from Mars as from the Moon (3800 m/s versus 1730 m/s). But on the other hand Mars has an atmosphere and while it isn't think enough for you to touch down gently with a big mass you'll only need 200 m/s or so to cushion your landing. But you need heat shields on Mars, but you need those to get back to Earth too. But lack of atmosphere means you could maybe use a mass driver to help take off from the Moon.
Mars has a gravity that's 2/5 of Earth's while the Moon's is 1/6. Is that important for human health? We have no frigging clue since the budget for a centrifuge on the ISS was cut.
The Moon rotates once every month and Mars once every 24 hours. If you're using solar panels it's a big deal making it though the lunar night, you'll need lots of batteries, maybe 25 times the weight of your panels. On Mars you need maybe the weight of your panels in batteries. There are the Peaks of Eternal Light on the south pole of the moon where this isn't an issue and you can just keep pointing your solar panels at different parts of the horizon all month.
And finally the martian atmosphere provides some protection against radiation and small meteors. It's not a super friendly environment in either case with no magnetic field so you'll probably be spending a lot of time under ground in either location.
So, it's complicated and I'm not willing to say that one is the true path and the other is stupid.
I'm sure it won't be good. But with a moon colony you can rotate people back to earth. Not practical with Mars.
The poles are one solution. Another is to put a large mirror in orbit to reflect power to the moon base.