I doubt the dropoff curve is linear. It's probably negligible up to a small threshold (500 ms?) with exponential decay in the tail. I'm willing to believe that a 1-second delay is a 10% hit, though I wonder how that would be measured, but this doesn't necessarily entail that 100ms costs 1%.
