Good point, I'm sure some people would argue that "no" is also pretty powerful. We are faced with so many decisions every day that our ability to say no is pretty important.
I would say that "No" - without hesitation, is far more powerful than Yes. Yes is expected, yes is what happens very frequently. But when "No" comes, we are taken aback, offended, and potentially have life-changing modifications to our self-esteem.
"Yes" keeps things going as they are - because with every question comes an expectation of "Yes" - if the many single men afraid of approaching a woman are any indication, most frequently, when we ask a question we expect a positive response - the times when we expect a negative reaction, the questions don't get asked.
For that reason, "yes" is - in my opinion, which I feel could be substantiated by data - not nearly as powerful as the opposite, no.