With regards to your first point - it's not really costing taxpayers in Wisconsin $4.5B . At least for the tax credits portion, those are taxes that would not have been paid in Wisconsin otherwise.
Kinda, but if no one offered such incentives, than Foxconn would've presumably still built its factory, and someone (maybe even Wisconsin) would've collected those taxes. So while for any particular state you can say that those taxes wouldn't have been collected, the general system of States competing for large businesses means someone somewhere won't collect taxes from Foxconn that they otherwise would've.
Plus, the taxes exist for a reason. Companies rely on public goods like roads, schools, etc for their business. So if Foxconn never came to Wisconsin, Wisconsin wouldn't have had to pay those taxes, but they also wouldn't have to pay for whatever extra public goods are consumed by the factory.
Of course, in reality, someone somewhere is going to pay for Wisconsin's schools, police, etc. So what really will happen is that Foxconn shareholders (who presumably don't, generally, live in WI) will have their share price subsidized, while other companies, or Wisconsin workers, or property holders or some combination thereof will make up the difference.
If you are outside of a tax jurisdiction, then yes, you shouldn't pay taxes inside that jurisdiction, because those are just taxes that wouldn't have been paid otherwise.
Are you paying 5+ digits tax and in the process of moving ? Then yes you can get the same special treatment while paying fraction of what you used to pay.
