Well there is an interesting history to passports. This article does a decent job of outlining how they morphed from a tool to help fight the Spanish Flu to what we see today.
https://apposition.substack.com/p/papers-please
Also there is quite a bit of economic work that showcases the positive impact of removing such stringent barriers to the free movement of people.
Of course, restricting people's movement across imaginary borders is very silly indeed. The point I was making is, I see plenty of frothing about vaccine passports, but not one single one of those people having conniptions about them is able to articulate what they would enable a government to do that they can't do already with your normal passport, driving license, tax ID etc.
The difference is that passports are only required for fairly rare activities (international travel, certain kinds of government interactions), whereas COVID passports are required for a lot of every-day things.
A driving license is a license is a certification that you've proven yourself capable to meet the minimum requirements to operate a potentially dangerous device, and more akin to doctor's license, lawyer license, etc. I don't think it's really the same thing.
My concern isn't really with "government can do $evil_stuff", but more in the general restriction of things.
This doesn't address the rather hysterical slippery slope argument that was being espoused. That's the one that I've never has a sane answer to - anything the government might want to do in the future with a vaccine passport, they can already do now with the plethora of unique identifying information they have on you.
