In my experience, if you're sorta close in English, you can sort of get your point across, even if you mispronounce words, because the mispronounced word means no sense so you start thinking about a possible explanation.
In Chinese, a different tone actually is a valid word with its own meaning, so it's a bit harder for you to get your point across.
To be fair though, if you nail all the other parts of pronunciation/speaking correctly and only get the tones wrong, a Chinese speaker will be able to infer your meaning the vast majority of the time.
Half of Korean vocabulary is Chinese, with pronunciation based on ancient Chinese pronunciation, but Korean has no tones and their writing system has recently become more or less completely phonetic. Disambiguation is still not a huge problem for them.
I've met foreigners who have had TERRIBLE tones but still manage to get their point across in Mandarin because of context. Your explanation for why English works despite mispronunciation is the same reason why Chinese works: context. For example, let's assume we were talking about our families. You want to say: 我爱我妈 (wo ai wo ma) "I love my mother". But if you said 我爱我马 (wo ai wo ma) "I love my horse", I would assume you meant you love your mother because of the context.
In Chinese, a different tone actually is a valid word with its own meaning, so it's a bit harder for you to get your point across.