Fwiw, you can optimize your height map / grid traversal  quite a bit. There don’t need to be any ceil/floor calls within the inner loop if you do a “3D DDA” iteration (like Bresenham’s for lines but in 3D).
We used to have good slides on this from Steve Parker’s CS 6620 at Utah, but modern versions of the class seem to use OptiX instead. Some folks put their code up online, so if you search for CS 6620 and height map or terrain you might find it. Useful for volume rendering too!
These beautiful 3D shaded relief maps have been going past in my social feeds recently - https://scottreinhard.com/Mapping-and-Visualization - seems like your tool could be used to make something similar without requiring familiarity with 3D modelling packages (Blender I think).
One of the main motivations behind developing rayshader was to democratize mapping--forcing people to become proficient in not one but two highly specialized software suites (any GIS software + blender) really made beautiful mapping unattainable for most. Plus, I just dislike GUIs in general, and wanted a code-based solution for map making :)
It would be cool to combine this with some kind of classifier for surface types and simulate water, glass, reflections, sub-surface scattering (basically other types of materials).
Have you played with any of the new hardware raytracing API’s from Nvidia RTX stuff?
While at Mapzen (RIP), I helped generate these from a wide selection of open elevation data sources.