I often feel the same way, although I’ve learnt to react quickly with an answer slightly better than ‘I don’t know’. Answering with related principles that you believe in and commenting on what you can, followed by ‘I need to think about this some more’ comes across better than ‘I don’t know’.
I wonder if this could be related somehow to the Dunning-Kruger effect? https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effec...