In fact, how about implementing a Logo turtle graphics system :)?
In general, ODE solvers can't be expected to maintain invariants of the ODE system, such as periodicity. This is typically handled by choosing a solver that maintains such invariants by construction, e.g., symplectic methods for Hamiltonian systems conserve symplectic 2-form. Adding many floating-point numbers together causes serious problems only in relatively rare cases, unlikely to occur here (e.g., see Higham's accuracy and stability book).
If I were to guess, the issue is maybe here (https://github.com/imeckler/diffgeo/blob/master/Main.elm#L45...), where the direction change is applied when the key is pressed, which means that the direction change doesn't happen inside the ODE itself: in effect this might be solving "go straight-tiny turn-go straight-tiny turn...", which would not be periodic even with perfectly exact solutions. If this is the case, this is like applying a first-order splitting method in which turns are handled with the forward Euler method, which would be quite inaccurate.
An extra source of error would be a positive Lyapunov exponent of the geodesic flow. When it is positive, the errors in the geodesic ODE solutions accumulate, causing exponentially large (in time) changes, meaning that the accumulated errors are much more noticeable than in geometries with non-positive Lyapunov exponents of geodesics (e.g., flat geometry would have smaller errors).
That said, it's not clear that you should actually expect to get periodic case in every case like this. I just kind of assumed they would be periodic based on how close they got. Still, flat circles seem to deviate after about half a dozen orbits. (Edit: With a bit of further experimenting, it seems curves in the hyperbolic upper half-plane that constantly turn should be periodic, so there's probably a bug.)
It might help to mod minecraft because players have a general idea of "how things should be" so you'd have a common frame of reference. No need to explain how flat world looks.
I guess the lines of curvature in e.g. the cylindrical geometry would look different.
Reminds me of a game that was featured recently on HN: HyperRogue. It's a simple roguelike on a hyperbolic plane.
 - http://www.roguetemple.com/z/hyper/
How do you scroll to zoom? Down arrow and Page Up/Down do nothing, and there's no scroll bar.
It's kind of interesting that moving while turning at a constant rate in the hyperbolic plane makes you gradually "drift". Is that actually true, or is it an artifact of the software?
However, this seems to simply be using the numeric library.
Which uses 'Dormand-Prince RK'.
Also, I notice that doing the same thing in the Klein model gives you very weird curves. Maybe you don't have the turning logic quite right? It should turn at a constant rate in the internal geometry, not in the plane geometry. (That shouldn't matter for the hyperbolic disk because it's conformal, so I guess something else is going on.)
Yet another feature request: support moving backwards