Easier said than done, but if you water-cool the GPU just upstream of your domestic water heater, it wouldn't be a bad thing. Perhaps using coolant and a counterflow heat exchanger, rather than the potable water, but the point stands. Would just need a secondary tank as a buffer (able to soak up heat from the GPU at all times, even when there's no demand for hot water) which then flushes in when demand exists.
Let's say your GPU uses a constant 300W. It takes approximately 1.16 Wh to heat a liter of water 1 degree celsius under 100% efficiency. This setup won't be 100% efficient, so let's round up to 1.5 Wh per liter per degree (roughly 80% efficient). A typical domestic water heater in the US 40-50 gallons, so roughly 160 liters.
Let's say your incoming water temperature is 18C and you want it preheated to 50C, which is 32C degree differential, which means you'll need 32 * 1.5 * 160 = 7680 Wh, or 25 hours straight to heat the buffer tank from scratch.
You'll need to purchase a small water-to-water heat exchanger ($50), two pumps ($100 each), a power supply for said pumps, hose and/or copper pipe and fittings, and various other sundries, plus the cost of a buffer tank ($600ish), so figure all in roughly $1000, plus the cost of electricity to run the pumps.
At $0.22/kWh you're saving roughly $450 a year with this setup in foregone water heating, but because it's not 100% efficient you're spending $500 in electricity to run your GPU 24/7/365, and that's the maximum you can possibly save with the above assumptions. Scale up for more GPUs and down accordingly for less usage as you see fit.
Alternatively, use air as the heat conductor by placing the GPU laden machine in the same space as a hybrid heat pump water heater.
Just recycle an old water heater as the tank? I think your cost estimate is way overblown but it would certainly eat up a lot of time and effort to DIY. Regardless there's a much bigger flaw with this plan. Water tanks need to be kept above or below certain temperatures. The "preheated" range being described here is distinctly unsafe due to various microorganisms IIUC.
