In most cases that setup does not absolve you from tax obligations where you live.
I can see a benefit in setting up a business or branch in EU in general, if you have customers in EU. Otherwise why Estonia specifically?
Because the Estonian government is fairly unique in that it will issue an electronic identification card to anyone who requests it and passes a background check, permitting that person--regardless of nationality--to use most Estonian government services, like business registration and tax filing, from anywhere.
I suppose, but only in the same sense as governments around the world tend to have mechanisms for non-residents and non-citizens to transact business or own property. An Estonian identity certificate doesn't, on its own, allow someone to enter Estonia or live or work there. That still requires citizenship of Estonia, the European Union, or meeting other requirements.
Anyone who wants to can accept the card (technically, the certificate stored on the smart card). For example, I use a domain registrar who registers .ee domains--thus requiring a digital signature from an Estonian ID--and can also use the card as a login mechanism. To be clear, the card isn't an ID in the sense of "here's my ID in person" (the card doesn't have a picture); it is a smart card with two digital certificates stored on it.
More to the point, though, is that Estonia provides a digital identity and signing system that, among other purposes, allows you to access Estonian government services like forming a company. Since you can form a business under Estonian law, you operate as part of the European Union single market.
With an Estonian (electronic) ID card, you can incorporate and run the business online. An Estonian corporation is permitted to do business anywhere in the EU.