The history goes farther back than the Civil War, and I don’t see strong evidence that it’s always racial in its origins.
A lot of immigrants, both recent and historical, came to the USA because they didn’t like the government in their home country, Pilgrims as a key example, Falun Gong members leaving China as a more recent example.
The general distrust of a strong government is still part of American culture.
A lot of people fought very hard for a century to try to prevent national government from being able to tell them they couldn't have slaves.