The British public think a good trade deal with the EU is more important than cutting immigration when negotiating Brexit [0].
Remember that about 20% of the leavers are 'liberal leavers' - they are pro-migration. Once you add this group to the remainers it's clear that a minority of anti-immigration voters shouldn't decide politics for everybody else.
Not sure how you can argue that there is no "actual public interest" in the topic.
See: "All Out War" by Tim Shipman for more on this - a really excellent book.