This is your perception, but if you don't use semantic HTML, you are building elements that look like something (for sighted users) but in reality, is something else (for assistive technologies and search engines).
So, it only works for a part of your users. And maybe it works for the majority of your users, but that means that you are making it hard for people that use assistive technologies and search engines to navigate your site.
Leave alone people with disabilities because you don't seem to care. Search engines need semantic HTML to crawl correctly your website, so it's an important part of any project.