The OP described the original offline PIN system. My memory is a bit fuzzy, but my recollection is that the original ATM PIN system was designed for offline verification and didn't allow user-selectable PINs. There was a tamper-resistant chip that encrypted the account number with single DES (using a system-wide fixed key) and took the first 16 bits of the result. There was a 16 entry table that mapped nybbles to keypad digits, and this was used to non-invertibly map the 16 bits nyble by nyble to a 4 digit decimal number. The interface to the tamper-resistant hardware module took in the 4-digit PIN and the account number and spat back out pass/fail, to minimize the utility of stealing an ATM. I probably got some of the details a bit wrong, but that's the gist of the system.
Soon after rollout, enough customers complained about wanting to set their own PINs that they added a 4 digit offset field to the magnetic stripe. The offset from the stripe was added to the user-entered PIN, and the least significant 4 digits were sent to the hardware verifier. That way the hardware verifier didn't need to be changed. The only way to change the PIN was by physically bringing the card to the bank. As I said, the details are probably a bit wrong, but it went something like that.
I read about one attack where the 16 entry nyble-to-digit mapping wasn't authenticated very well, so attackers could steal an ATM, then trick the hardware module into loading a mapping that mapped 0-7 to "0" and 8-F to "1", then bruit force the PINs for a bunch of cards. Then they'd change the mapping to 0-3 -> "0", 4-7 -> "1", 8-B -> "2", C-F -> "3". After two more rounds of this, they could use the production mapping of nybles to keypad digits. For each round, it takes an average of 8 guesses and a maximum of 16 guesses to guess the PIN with the PIN. (Information gained from one round is fed forward to the next round's guesses.) There are four rounds, so an average of 32 guesses and a maximum of 64 guesses to extract a PIN. Once the PINs were bruit-forced using the stolen ATM, they could be brought to a real ATM, even though the attack wasn't able to extract any information about the DES key used by the system.
A later system had a tamper-resistant module that used 3DES to encrypt modem traffic between the bank and the ATM. Changing 3DES keys involved two different employees being physically present at the the ATM. Each employee would load their key-change info and enter their PIN. A combination of a master 3DES key and the employee's PIN would be used only inside the tamper-resistant module to decrypt the key change information. The key change info from the two employees would be XOR'd together to get the new modem 3DES key and a keyed cryptographic message authentication code (MAC) over the new modem 3DES key. Only if the new MAC checked out would the new modem 3DES key replace the old modem 3DES key. This way, no one employee ever had enough information to learn the new 3DES modem key, even if they somehow stole the master 3DES key used to encrypt the key change information. As I said, I may have had the details a bit wrong, but it's none the less very well thought out.
These days, I would hope ATMs use something similar to Kerberos tunneled over TLS with pinned certificates so that just breaking the public key algorithm or just stealing the Kerberos shared secret isn't sufficient to read the traffic or spoof communications. The bandwidth required for an ATM is so low that doubling the encryption overhead vs. TLS isn't a big deal.
Anyway, I'll try and dig up the paper on the attack on the poor authentication of the nyble-to-decimal mapping in the tamper-resistant chip. It was quite an interesting read. I'm pretty sure the paper was the main subject of a Slashdot article in the early 2000s.
The published attack [1] used a more sophisticated pattern of compromised decimalization tables, recovering the PIN in an average of 15 instead of 32 attempts. The naive pattern I described will work, but the published attack is more than twice as efficient.
Also, in the IBM 3624 + Offset PIN block algorithm, the offset is subtracted (digit by digit without carry/borrow) from the customer-entered PIN, rather than added. [2]
My memory has faded a bit in the 12 years since I read the paper.
Soon after rollout, enough customers complained about wanting to set their own PINs that they added a 4 digit offset field to the magnetic stripe. The offset from the stripe was added to the user-entered PIN, and the least significant 4 digits were sent to the hardware verifier. That way the hardware verifier didn't need to be changed. The only way to change the PIN was by physically bringing the card to the bank. As I said, the details are probably a bit wrong, but it went something like that.
I read about one attack where the 16 entry nyble-to-digit mapping wasn't authenticated very well, so attackers could steal an ATM, then trick the hardware module into loading a mapping that mapped 0-7 to "0" and 8-F to "1", then bruit force the PINs for a bunch of cards. Then they'd change the mapping to 0-3 -> "0", 4-7 -> "1", 8-B -> "2", C-F -> "3". After two more rounds of this, they could use the production mapping of nybles to keypad digits. For each round, it takes an average of 8 guesses and a maximum of 16 guesses to guess the PIN with the PIN. (Information gained from one round is fed forward to the next round's guesses.) There are four rounds, so an average of 32 guesses and a maximum of 64 guesses to extract a PIN. Once the PINs were bruit-forced using the stolen ATM, they could be brought to a real ATM, even though the attack wasn't able to extract any information about the DES key used by the system.
A later system had a tamper-resistant module that used 3DES to encrypt modem traffic between the bank and the ATM. Changing 3DES keys involved two different employees being physically present at the the ATM. Each employee would load their key-change info and enter their PIN. A combination of a master 3DES key and the employee's PIN would be used only inside the tamper-resistant module to decrypt the key change information. The key change info from the two employees would be XOR'd together to get the new modem 3DES key and a keyed cryptographic message authentication code (MAC) over the new modem 3DES key. Only if the new MAC checked out would the new modem 3DES key replace the old modem 3DES key. This way, no one employee ever had enough information to learn the new 3DES modem key, even if they somehow stole the master 3DES key used to encrypt the key change information. As I said, I may have had the details a bit wrong, but it's none the less very well thought out.
These days, I would hope ATMs use something similar to Kerberos tunneled over TLS with pinned certificates so that just breaking the public key algorithm or just stealing the Kerberos shared secret isn't sufficient to read the traffic or spoof communications. The bandwidth required for an ATM is so low that doubling the encryption overhead vs. TLS isn't a big deal.
Anyway, I'll try and dig up the paper on the attack on the poor authentication of the nyble-to-decimal mapping in the tamper-resistant chip. It was quite an interesting read. I'm pretty sure the paper was the main subject of a Slashdot article in the early 2000s.