Yeah, they're not sensitive to magnetic fields. But they do drift with temperature. Which leads to fancy semi-improv solutions, like placing the quartz in a small case, together with a heating element and a temperature sensor, to keep that sucker incubated at constant temp. A lot of HAMs used to do this back in the day - or maybe they still do, I dunno.
Also, placing a small variable capacitor in series with the quartz is an easy way to tune it, if it's too fast. I actually have a wall clock that looks nifty but is too fast, and I'm gonna tune it by trial and error, inserting small caps in series with the crystal.
The heating setup is still used today. In fact, you can get integrated components called OCXOs (Oven Controlled Crystal Oscillator) that do the whole job in one small package with precision over time on the order of 1ppb. Cheaper, and less precise, are TCXOs (Temperature Compensated Crystal Oscillators) where a crystal is paired with a circuit that has an opposite dependency on temperature so that the errors tend to cancel each other out.
Also, placing a small variable capacitor in series with the quartz is an easy way to tune it, if it's too fast. I actually have a wall clock that looks nifty but is too fast, and I'm gonna tune it by trial and error, inserting small caps in series with the crystal.