Depends on where you are. Where I live, it's cold seven months out of the year so keeping the resultant heat inside the house lowers the heating bill somewhat.
I've always wished for a fridge that was mounted to an exterior wall. In the summer, it operates like a normal fridge but with the condenser coil outside of the house. In the winter, the compressor system turns off when the outside temp goes below a certain value and little door/shutters with fans regulate the temperature of the refrigerator either by letting in cold air from outside (if the interior needs to be colder) or from the inside (if the interior needs to be warmer to prevent food from freezing).
There's probably a few good reasons why this is not actually practical but I'm tempted to try something like it in retirement.
Aside the complexity and standardization needed for building it, which alone will absolutely not make it worth it, you'd also have the problem of temperature fluctuations. You probably don't want your milk to be frozen right? ;-)
So if you really need to push indoor air inside to avoid that, you are basically killing the efficiency. And not only that, you would push warm and comparibly humid air inside the fridge a lot. It will condense and now your fridge has to deal with a lot of condensation and probably get moldy.
But good to see that I'm not the only one making these thought experiments.
My personal conclusion is: eventually we will have a single refrigerant distribution system where other systems can be connected and then make use of it (similar to multi-split ACs, but extended to more devices). Then, your fridge would just be part of the system.
And, just to understand: transferring heat from inside the fridge to the outside in the case that the outside is colder is so energy efficient already, there's no need at all to optimize anything. A solar panel as big as your hands will be sufficient even in winter to take care of keeping the fridge cool. That's how little energy is needed.
An energy star fridge uses about 400kWh/year. In pittsburgh, we have 5710 cooling degree days and 736 cooling degree days. Those cooling degree days are spread across about 4 months. So we only need to dump heat from the refrigerator about 1/3 of the time: 133kWh. With an A/C CoP of 5, that means that cooling the extra load from your fridge costs you about an extra 26kWh / year, or about $5.
On the heating side, it's actually better to heat the house from the fridge than to burn gas - you've basically got a heat pump system! Heat pumps are more efficient than direct heating. If you already have an efficient heat pump system this may not be as true but it probably comes out in the wash.
The physical modifications to the house would have to be very cheap to make this cost-effective -- and more penetrations through the house envelope create opportunities for leaks.
I've always wished for a fridge that was mounted to an exterior wall. In the summer, it operates like a normal fridge but with the condenser coil outside of the house. In the winter, the compressor system turns off when the outside temp goes below a certain value and little door/shutters with fans regulate the temperature of the refrigerator either by letting in cold air from outside (if the interior needs to be colder) or from the inside (if the interior needs to be warmer to prevent food from freezing).
There's probably a few good reasons why this is not actually practical but I'm tempted to try something like it in retirement.