I'm assuming you are still losing power. Some power is taken out via the USB and put back into the computer, and some is use to power the device like normal. Since the power put back into the computer is not larger than the power being taken out, there is still a net loss in power over time.
The problem is not to do with the 'power put back' being 'larger', it's to do with efficiency. If charging and discharging a battery were 100% efficient and there were no losses to heat through the rest of the system, then there'd be no discharging, but, yes, due to the laws of the universe, this will always cause a net loss.