If you don't have something to hide, your life is lame. LOL
Seriously, though, what would the HIPAA violation be for location data? Knowledge of someone going to a doctor's office doesn't sound like a HIPAA violation. AFAIK, violations only relate to what is communicated between doctors (and other healthcare professionals) and patients.
Location data generated by your phone is not covered by HIPAA (source: [1]) whereas the location of a patient undergoing treatment is. Thus, there's nothing that stops a data broker inferring that you are visiting a psychiatrist or a reproductive health clinic and sharing that insight with buyers, but the clinic/doctor cant share that you were treated at such and such location since that is personal health information (PHI).
The web page below has quite some discussion on what this means for patient privacy and how to disable certain location services on your phone.
Seriously, though, what would the HIPAA violation be for location data? Knowledge of someone going to a doctor's office doesn't sound like a HIPAA violation. AFAIK, violations only relate to what is communicated between doctors (and other healthcare professionals) and patients.