You're thinking in terms of a steady-state. If a middle-man is able to do such a high markup then it means that yes we need more middle-men, because the increased competition will reduce their ability to have such markups. The need is proven by the high margin itself. If we add more factories and factory workers, on the other hand, end buyers won't gain squat since you're only addressing a tiny component of the cost.
