Madre is used more often in phrases, such as "Madre de dios!", Or "La madre putria," or "Pinche tu madre", or it's used as a formal to refer to one's mother. It's similar in English, such as "On my mother's side", "Mother fucker", "Mother of God!", etc.
In Mexico, "tu madre" can be a general insult, probably shortened from "pinche tu madre", though in English we have a similar vulgar retort of "your mother". As a weird turn on the phrase, "de puta madre" is slang for a good thing.
I'm not aware of an association with brothels in Mexico, but certainly in some Latin American country it could have.