"If all members of a society were perfect bandits the society would remain stagnant but there would be no major disaster."
There would be no incentive to actually work, as everything presumably would be stolen. So I don't think stagnation would be the result. A steady decline back to hunter gatherer seems more likely.
There are no perfect bandits, let along a society full of them. Scenarios like "If all members of a society were perfect bandits" are hypothetical and don't really translate to any actual human society that you could imagine. Even so, they're useful for illustrating concepts.