With a virtual instrument you would usually not face the full round trip latency, but rather control latency (MIDI?) and audio output latency (usually 50% of roundtrip). So in the VSTi case that might be closer to 25ms than 40ms in practice. Often the latency jitter is quite dreadful in this case as the control signal is not synchronous to audio.
Did you ever see people who try to sing with headphones with a delay of around 200ms? That really messes up your performance.