It's not if no-one cares, it's if not EVERYONE cares.
If you've got 10 people in the meeting and 6 people start discussing something, you've got 4 people who are having their time wasted and were given no choice in the matter.
Perhaps it has to do with the attitude towards meetings. If someone is wasting his/her time in a meeting, he/her should excuse him/herself and leave.
It's the same pattern as when you need to be in a meeting about a specific point, but not attend all the other points. You ask to have the stuff you care about come first and you can leave it after. If some unexpected things needs your input, you'll be called in anyway.
I'm not sure if that's as efficient as sticking to the prescribed agenda, and having the non-agenda stuff taken offline / discussed after.
You don't have to schedule an entirely different meeting room to talk about it - more often than not at my work, the people who want to discuss the unexpected point just stay after the original meeting has ended to discuss it.
To everyone who was there for the agenda, the meeting is effectively over - for the others, you could say the meeting continues on with the unexpected topic.
The important part is that everyone meets for as short as is needed - they don't have to keep running in and out of meetings to see if things are relevant, or stay and listen to things that don't matter to them.
If you've got 10 people in the meeting and 6 people start discussing something, you've got 4 people who are having their time wasted and were given no choice in the matter.