You wouldn't lock the account forever, the point is to establish that the person whose password was compromised knows, that the password is not the only factor which is used to regain access to the account, and to ensure that your service (rubygems) and its downstream users are not compromised as well as a result of the breach.
Any groaning about the inconvenience caused by disabling account access until the password is changed, can be simply shrugged away in favor of security concerns, with a link to this story about rest-client.
By the time you have learned the user's plaintext password, their account may already have been compromised. There's a case to make that you disable all downloads of any gems that might be compromised from the account until you've verified they aren't. That might be over the top, especially for popular projects as now we are talking serious inconvenience affecting potentially thousands or more of downstreams.
It's a sticky situation, since you don't really know how long that password has been in the open for hackers to use and abuse once you've discovered it in a password dump.