Or you could just explain that fair and beautiful refer to the content of their character and swarthy is a synonym for smarmy, or evil and has nothing to do with skin color at all.
Pretty much every letter Tolkien wrote that touches on racial issues was quite anti-racist. He condemned the nazis, and yet also condemned racist anti-german propaganda in England during the war. He also expressed sympathy with the treatment of black people in South Africa in a letter to his son Christopher.