I thought you were making a distinction between quoting in fractions of a dollar vs quoting in pennies. It isn't really material. If you quote in fractions but allow the minimum price variation to be 1/128th of a dollar you have a market that allows for tighter spreads than one in which the minimum price variation is a penny. There is nothing wrong with prices in fractions of a dollar. What matters is the minimum price variation a.k.a. the tick size.
It is material, because the fractions of dollars we're talking about were bigger than 1/100th of a dollar. Clearly, they could have moved to sub-decimal fractions, but the point isn't "fractions" versus "decimals"; it's "human market makers fixed the fractions at 1/8ths and 1/16ths of a dollar to artificially increase the spread", versus "algorithmic trading competed the spread down below a penny".
You implied that maybe humans had been using fractions because they were easier to deal with. No. They were using the fractions they were using in order to skim extra money off trades, and they fought tooth and nail to prevent competition from reducing spreads below 1/8ths --- in fact, if you read the link I showed you, it wasn't even 1/8ths; it was 1/4s!
What I meant to imply was that the emergence of fractional dollar tick sizes was probably due to the fact that they were easier to deal with before the advent of computerized execution, clearing, and settlement. By the mid nineties most of that had been automated and there was therefore no good reason for such large tick sizes. When humans were still doing all trade processing manually, it made sense to have a system that incentivized fewer large trades (vs more smaller trades). Large minimum price variations and round lot requirements would have been a means to that end.
