You literally can't do anything useful with Gemini (as a Google Assistant replacement) without granting it access to everything. And I mean that in the most literal sense possible. It doesn't know the weather, the time, it can't play music, nothing.
Taken in the most literal sense possible, if you're using Gemini in any capacity today, it's authorized to use your photos to serve you personalized ads.
Funnily it is against GDPR: one must not limit any service if they can be solved without the given data not allowed to be accessed. Of course this is vague enough, but EU has given big fines, thanks for Google's donations to this continent in 5 years.
Taken in the most literal sense possible, if you're using Gemini in any capacity today, it's authorized to use your photos to serve you personalized ads.