I don't think you have to take Musk at his word to see that he's not going to get people on Mars without significant resources that earthlings would prefer go towards other things. Government is likely to become a problem for him once the impacts of those decisions are felt.
To him, government efficiency means that the government is out of the way and no longer interfering with his efficiency. Might as well just do away with it entirely.
People like Musk dislike the government because people have a say. That say is imperfect and often corrupted, but people get to have some amount of input. They can vote, and organize, and they could decide that they'd rather have roads or water or any number of silly things that get in his way.
Companies are not democracies, Musk says "get rid of the supercharger team" and no one gets a vote, they just get rid of them. We should all be worried about what goals he efficiently wants to achieve, if you or I were in the way, would he care.
The danger is in the unchecked nature of this power, even if someone likes Elon Musk and thinks he's a brilliant genius, what gives him the right to supplant the will of the people with his own.
Elon has shown time and again that he will prioritize what HE wants, and if that means some people don't have jobs, well that's just fine. If that means that people should have to sleep on the factory floor and wake up and make cars he can profit off of and then back to sleep on the factory floor, that's also just fine. If that means that USAID doesn't feed the hungry, that's also just fine.
And maybe to anyone reading this that's happy with what he's done, you are just fine with it too. But what happens when he decides something you do care about is alright to destroy too to meet his goals, he's just fine with it. What do you do then? How confident are you that your goals and his will stay aligned, forever, that the ax of "efficiency" won't come for you and yours someday. And if that day comes, what will you say to the people that tell you "he's a brilliant genius and he's fine with it so so am I"
Even people who love Musk need to realize they're cogs in his machine. He's not here to make your life better, he's here to make sure nothing is in the way of the things he wants to achieve. He doesn't care about the quality of your life, the only thing you have, he cares about how much of it can be captured to his ends.
For some people, that seems to be fine. They work at SpaceX and Tesla and his other companies. For many of us, life is the journey and the quality of it matters, not the Martian destination, and he can fuck right off.
If the issue were actual waste and balancing the budget, they'll go after the _actual waste_ like the military budget, or Trump's golf outings. They wouldn't decrease taxes, especially not at the top.
Also trust is earned. Do I have to remind you of the hundreds or thousands of lies, many of them very recent, of these people?
The goal is to go after waste, fraud and abuse.