I have a verizon LTE device that I tether with from time to time, and the ping times I get to my home server are insane, usually around 20ms. When I tether on my HSPA galaxy nexus, ping times are usually around 100+ms.
Now this could be verizon vs t-mobile and not LTE vs HSPA (or some mix). Verizon is the only/main LTE-supporting provider right now, so LTE and Verizon are conflated in my mind, for better or for worse.
Do you have real-looking latency data for the first few hops from your phone? (...keeping in mind that latency information for intermediate hops is not always accurate.)
100ms vs 20ms could be t-mobile peering with your home ISP in some far away city, while Verizon peers locally. Based on only pings, you'd never know.