One thing they could do to help alleviate this is favor the Unicode section(s) associated with the visitor's language, as well as punctuation. That way we'd be less likely to get results like "Ogham letter nion: ᚅ" for drawing pi.
I tried the ae ligature, long s, sharp s, Russian zha, Greek capital gamma, and a heart, and of those only the gamma was in the top five (it was #2); zha, ae, and heart weren't even in the list, and long-s and sharp-s were waaay down the list. (Actually, I just tried it with a better-drawn sharp-s and it didn't even turn up.) In some cases the top choices really do resemble the letter I was looking for, so that's fine, but there's a lot that shows up that resembles the canonical form a lot less than my drawing does.
So, not quite ready for prime time, I think. Cool idea, though.