Your thesis is at odds with the experiences of others. For example, Marissa Mayer from Google once noted that a 500ms delay in search results directly resulted in a 20% drop in traffic. And Amazon's A/B testing indicated that even a slowdown of 100ms resulted in a measurable drop in sales.
Speed definitely matters.