It's still a legitimate concern. If a person or entity does something that flagrantly violates Visa's terms of service, Visa can refuse to process payments to them. If that same entity gets 86'd by 2/3rds of the Libra Alliance, they could be blocked from participating in transactions, which would make their Libra balance essentially worthless.
