If you believe that the government cannot restrain itself, I think you'll have difficulty accounting for that basic historical fact.
Therefore I am concerned about government growth and encroachment, because it takes decades and the blood of heroes to recover lost freedoms, and by then I'll be old or dead. Better to be concerned now and look foolish than to dismiss concerns and be proven naive.
The only way I could really see tax rates being a top-tier freedom issue is as a restraint on government: the Grover Norquist theory that low taxes starves the government of the ability to do things, so in turn reduces spending and government involvement in everyone's lives. But I haven't seen that over decades. We lowered taxes in the 1980s, and that didn't stop government spending from more than doubling in that decade. We lowered taxes again in the 1990s and 2000s, so the government couldn't afford to invade Iraq or Afghanistan, and we invaded both anyway, borrowing money we didn't have to spend on the invasion (and we're still spending money we don't have on both), and then we spent more money we didn't have on bailouts in late 2008. Basically it seems over at least 70-80 years or so, tax revenue hasn't really been a meaningful constraint on spending.
I take tax-rates much more seriously. Taxes are a direct line to slavery--obviously a 100% tax rate would be complete slavery. My current 1/3 tax rate makes me 1/3 slave--maybe a fair trade for being 2/3 free, but a trade nonetheless. Unfortunately, as you succinctly pointed out, tax rates of late have had little relation to spending, but in the long-term input = output, so as a young individual I see taxes and spending as two sides of the same coin.
To the extent our government has shown restraint, I think that is due to people making their concerns known and visible, as well as some good old fashioned American luck. Politicians will tend to acquire as much power as the people let them.
What exactly does concern accomplish? Shouldn't you be "concerned" about space aliens invading Earth? Shouldn't you be concerned that your neighbor is a witch who has put a hex on you? I mean, it would much better to be "concerned" about that dire threat than to merely look foolish for ignoring it, right?
This rhetoric about "concern" troubles me because it comes from nowhere. It is not based on a factual analysis about the growth of government power in recent history; it is just ahistorical venting of misplaced anxiety.
The fact of the matter is that in the US, top marginal tax rates have declined significantly over the last half century. Government control of industrial policy has declined radically. Government commissions used to literally set prices for all kinds of goods. Airlines couldn't set their own routes and schedules while banks couldn't open branches in adjacent states. Now, all those regulations are not only gone but considered quaint and absurd. Does any of that history ameliorate your "concern" at all? Or is your concern resistant to facts?
To have my concerns about governmental encroachment on freedoms put on the same level as space aliens does put us on a different planet, philosophically.