Because it employs more people and pays taxes locally. It's perfectly valid for the government to ignore that and focus solely on the consumer side (i.e. minimize prices), but it's also perfectly valid (and economically sound) to do what France is doing.
>What is the implied incentive in this case? It seems like it's just incentivizing a brick and mortar business model
No, it is not incentivizing it. It levels the playing field a bit by applying more consistent taxation.
>Doesn't this all point to a fundamental problem with "corporate tax"? Wouldn't it be better (fairer) to reduce or remove corporate tax and raise sales tax or similar, such that it doesn't matter whether the business has a presence in the country in question?
I can be persuaded by that argument (economists do love consumption taxes for a reason, after all), but either way it's not for Amazon to decide. In France, it is for the French people to decide, via their representatives. Whatever positive or negative consequences arise are also theirs to bear.
Well, now you run into the problem of progressive vs regressive taxation. Since consumption taxes, as the name implies, tax consumption, and the poor have a marginal propensity to consume of almost 1.0 while the wealthy can have an extremely low marginal propensity to consume, it's reasonable to conclude that consumption taxes are taxing the people who are least able to pay and who would already be spending that money anyways. You're taking away money that would already have stimulated the economy elsewhere.
Taxing the poor runs into these kind of zero-sum problems much more quickly. The rich tend to save or are more able to engage in activities which produce products of much more questionable economic value (i.e., bitcoin farms, copyright trolls, rent-seeking, etc.). Obviously investment has value, but in economic terms it may or may not generate a commensurate amount of wealth.