Doing nothing but an XOR is usually fine but one thing to keep in mind is that an attacker that can see one stream and control the other could completely eliminate any entropy yet make the data look completely random.
This sounds like some unlikely scenario but for instance the Linux kernel uses a method like this for /dev/random. Entropy is collected and mixed in an entropy pool from many sources but at the end it is XORed with the output of Rdrand on processors that support it. The NSA could force Intel to sign a malicious microcode update that changes Rdrand to AES_encrypt(i++, NSA_KEY) ^ entropy_pool and then any random data coming from the kernel is completely predictable and without NSA_KEY it wouldn't be discernible from truly random data.
That's not really right. The other requirement for that scheme to work is to leak the entropy_pool as well. That's why the linux kernel random source IS safe from bad hardware. Even if the hardware source is completely known by a third party, they can't discern the output from random data because it has been XORed with random data.
If it helps to think about, imagine that the evil hardware always output 00000000 for a byte of randomness. The kernel then XORs that with a byte from the entropy_pool, which is unknown to the adversary. The output byte is still completely unknown to the adversary even though it knows one of the inputs.
This sounds like some unlikely scenario but for instance the Linux kernel uses a method like this for /dev/random. Entropy is collected and mixed in an entropy pool from many sources but at the end it is XORed with the output of Rdrand on processors that support it. The NSA could force Intel to sign a malicious microcode update that changes Rdrand to AES_encrypt(i++, NSA_KEY) ^ entropy_pool and then any random data coming from the kernel is completely predictable and without NSA_KEY it wouldn't be discernible from truly random data.