I've variously heard it was a 5% commission for the groom, jockey, or auctioneer. If a horse won a 100 guinea prize, the owner conventionally got 100 pounds and someone else got 100 shillings in commission.
I don't know about this aspect, but horse sales typically involved this type of commission. The buyer would pay in guineas (or £1.05 times the nominal purchase price) and the seller would receive that number of pounds, with the extra going to the auctioneer.
I'm not sure if this system persists now, but it definitely lasted long past decimalization.
