It seems like the bigger issue is when malicious parties usurp this data and use it for their own ends. We can look to science-fiction for predictions. Scroogled comes to mind, as well as a (possibly falsely remembered) recent campaign IRL to root out dissidents via internet searches.
And one more scenario: false flag. Since search records are considered evidence, planting this evidence could be all the sway that a jury needs, or even just for defamation purposes.
I think the fear is not of Google, but of making our lives tied to a record-keeper that could potentially have no accountability.