The loading time is due to it having to load the elevation data, although it will be cached on subsequent visits, and it is further slowed down by client side processing, which is currently done in JS, but in the future will be hardware accelerated
As for the controls, do you find it more difficult than something like Google Earth? I was hoping that people used to that would be at home with the controls. What exactly would you expect the arrow keys to do? Simulate a mouse movement in that direction by a constant?
As for instructions, were there any parts in particular that were tough to figure out. I think one needs to strike a balance between having no hints and pointing out the blatantly obvious.