Your subject says "is it costing me users?", but your message says "how many people will be unable to access the site?".
These are two different things.
If your site is slow, it will cost you users. v6 has measurably better performance for websites, so not having it will cost you users. Those users will still be able to access your site though, they'll just choose not to because it's too slow for them.
The practical reality is that almost all client devices that are used for browsing the web will have some mechanism to reach legacy v4-only websites (but the mechanism might itself be a cause of poor performance...).
On the server side, v6-only servers (VPSs/containers/etc) are getting somewhat more common though, so if you're hosting something that's expected to be downloaded directly onto a server then please figure out v6 somehow.
These are two different things.
If your site is slow, it will cost you users. v6 has measurably better performance for websites, so not having it will cost you users. Those users will still be able to access your site though, they'll just choose not to because it's too slow for them.
The practical reality is that almost all client devices that are used for browsing the web will have some mechanism to reach legacy v4-only websites (but the mechanism might itself be a cause of poor performance...).
On the server side, v6-only servers (VPSs/containers/etc) are getting somewhat more common though, so if you're hosting something that's expected to be downloaded directly onto a server then please figure out v6 somehow.