You are right that your choice of features plays into it, but I don't believe that's all there is. Even in the best case (all of features you want are available in all the browsers you care to support), you still should be testing in those browsers (ideally on all of the platforms / devices you support).
You also have to make the initial decision about what browsers/platforms/devices you support in the first place, and then choose when to re-evaluate your support. None of that is free.
Please don't get me wrong: I think it's a fine path (and one I have chosen myself), but it's definitely not without downsides.