Actually paying French income tax is pretty easy; it's paying US taxes if you are a US citizen working in France that's hard. I'm not sure Americans know that they are taxed on their income wherever they earn it. All other countries only tax your income that you earn in that country. French citizens don't pay income tax on their salary they earn in the US, but the reverse is not true. Not only do Americans have to pay French income tax, they have to pay US income tax, even if they earn all their salary in France and stayed in France the entire year. And it gets complicated how to calculate, which is why I pay HR Block $700 to do it for me.
You're right about the US being one of the few countries in the world that still tax citizens and permanent residents no matter where they are in the world. You are also correct you should get some tax accounting advice to file your US taxes[]. Having said that, the US provides two mechanisms to offset US taxes when you are living abroad:
1) For single income earners, the first ~$100K is exempt or "non-reportable", known as the Foreign Earned Income Exclusion[1].
2) US tax treaties with other countries help to offset. Thus, if a US citizen/resident pays a lot of taxes in another country that would exceed or equal their amount in the US, provided the country has a reciprocal tax treaty, the person should not need to pay additional tax to the IRS[2].
Note that while this applies to US permanent residents too, there are immigration consequences if one lives abroad without the appropriate paperwork/authorization.[]
[*] Disclaimer: I'm neither a lawyer nor an accountant. Please consult at least the latter for your taxes.
I'm a bit annoyed about this. European forms always ask about the US, even if there's no reason to suspect I've ever been a US citizen - I assume this is purely because America has the pull?
It is true that US citizens must file their yearly US tax return, no matter where they live. That is different from the majority of other countries that do not require this from their citizens living abroad.
In practice, double taxation - which your comment seem to imply - is not really an issue. A lot of countries - including France - have mutual agreements with the US that exempt foreign income (up to a certain amount) from being taxed twice.
Under many circumstances, you can also receive a credit for taxes paid to another country, which ensures you pay max(local tax, US tax). I can't imagine French taxes are lower than the US, so...you end up paying nothing.