My father taught me to read when I was 2 and from that point on, everyone encouraged my parents to "fast track" me. I was tested, examined, and prodded by psychologists, doctors, teachers, and "experts". I even passed the preschool entrance exam before my older brother (he's been paying me back ever since lol).
Finally one day, my father, of all people, said enough. I would mainstream with all the other kids because he didn't want me to be a "freak". To this day, I really don't know if it was a wise decision or a snap judgement.
So I sat in class, bored to tears for the next 12 grueling years. Looking back, I had no choice but to "let my love of something pull me" as the OP said. So I learned a musical instrument, started several small businesses, made home movies, and published my own magazine. I excelled in everything outside of school and did poorly in class. I wonder what college admissions officers thought about a self-published C student with perfect SAT scores. I think my magazaine did more for my future that anything from school.
After a great college experience, I spent years of torture in corporate cubicles, bored to death no matter what the job was. Only when I found a way to do my own software startup, did everything fall into place for me. I'm finally living the life I was always meant to live.
So this would-be prodigy ended up being a late bloomer. I don't know whether this is better or worse, but I sure am glad I finally ended up where I belong.